Rich : MARS Datamanagement Overview

2 The mars2ssds "stuff" runs on pismo, basically there are three main tasks that scrape data via web requests from the pmacs server archives, break them into pseudo 'ssds devices' and publish to rabbitMQ as ssds protocol buffers. (psc2rabbit, onewire2rabbit, and npc2rabbit tasks in /opt/ssds/mars2ssds)

3 Qmonitor (/opt/ssds/qmonitor) run perodically by cron, (/u/ssds/crontab.pismo) looks at rabbitmq messaging server, whenever it detects an 'active' device queue with no consumer, it starts a consumer for it, that pulls protocol buffer packets and pushes them to sql server dione device base tables in SSDS_Data.

(Note: from home dir /u/ssdsadmin see special files crontab.pismo and .netrc Both mars2ssds and qmonitor projects can be checked out from svn repository in projects of those same names. The cronjob and .netrc are also tucked away for lack of a better place, in the qmonitor/bin dir, although they need to be deployed to /u/ssdsadmin on pismo.)

On Jul 25, 2012, at 8:19 PM, Rich Schramm wrote:

three separate projects are checked in to svn:  mars2ssds, qmonitor and mars2web
mars2web stuff goes to pmacs user acct on pmacs server. (see README.txt in the mars2web/bin dir)
*** everything else below refers to user ssdsadmin on pismo for the mars2ssds and qmonitor stuff ***

also the cronjob.pismo and the .netrc files are in svn in the qmonitor/bin directory

everything should restart on reboot or if the database is rebooted etc. - needs further testing to verify

run logs are in /opt/ssds/mars2ssds/log and /opt/ssds/qmonitor/log and are set to rotate.

TODO:
1)npc2rabbit has the external loads 1-8 turned off. We need to decide on in-row-storage and then run the bulk loads
of legacy data (by running npc2server.py), then turn on the real-time loads (uncomment the lines in npc2rabbit.py)

Future enhancement:
1)Would be 'nice' if onewire/psc/npc2rabbit.py would query for the max(timestampseconds)
for a device and pickup processing from there.... instead of attempting to reload
all records from midnight of the day it is started.  very minor issue as the insert just ignores records already loaded